Versatile video transformation device

ABSTRACT

A versatile video transformation device and adaptive image processing methodology thereof to digitally scan convert, that is, reformat TV raster scan video and particularly high definition (HD) and/or digital DTV (particularly those for example in 1920x1080i or 1280x720p format) video data and associated synchronizing signals, for the purpose of making present standard television sets compatible at low cost with the latest advancements in free HD whilst allowing a multitude of other DTV programs and ancillary data to fill to a greater extent the remaining channel allocations by FCC to TV broadcasters.

FIELD OF THE INVENTION

The present invention is directed towards versatile video transformationdevice and adaptive image processing methodology thereof to digitallyscan convert, that is, reformat TV raster scan video and particularlyhigh definition (HD) and/or digital DTV video data and associatedsynchronizing signals, for the purpose of making present standardtelevision sets compatible at low cost with the latest advancements infree HD whilst allowing a multitude of other DTV programs and ancillarydata to fill to a greater extent the remaining channel allocations byFCC to TV broadcasters.

BACKGROUND OF THE INVENTION

Present TV sets are not capable of using HDTV, but rather are limited toa fixed-format and are incompatible with different TV raster formatssuch as that provided by HDTV. While an amount of versatility (andenergy-efficiency) has been provided by U.S. Pat. No. 4,361,785, thedisclosure of which is incorporated herein, it is desirable to improveand expand it, particularly, with regard to standard TV sets.

The inexpensive utility of versatility for single frequency standard TVis especially desirable now because of the availability of HDTV and DTV.

In the past, when color TV broadcasts became available, existingmonochrome "black & white" TV sets, at the time, could not display theadded dimension of color. Viewers could, however, still see inmonochrome sets new programs broadcast in color because the novelchrominace signals conveying the hue and saturation (color purity)attributes were rationalized and compressed into the then existing videobandwidth without substantial interaction with the luminance. While thepresent invention is not directed towards improving the TV set circuitryper se, in so far as DTV system eliminates many visual impairments eg.interline flicker, sync jitter, moire, raster line structure, multipathghosts, chrominace distortions etc. the picture quality of such TV setsshould be markedly improved. This will be accomplished while, mostimportantly, addressing the broadcaster's desire to allocate, as much aspossible, other programs and data communications within the channelassigned to it by the FCC.

In this regard, the FCC has allocated the traditional broadcasters a newchannel for DTV which can include "multicasting" several simultaneousprograms with conditional access for a nominal fee that would offset thecosts of new equipment needed to broadcast DTV and free HDTV and alsoprovide better competition with costly cable TV and satellite TV.

Ironically, the effective expansion of TV channel bandwidth capacitywrought by removal of redundancy in video compression techniques forHDTV appears directed towards the of viewers' desire to receive morechannels due to the quality of programming, or lack thereof. Theimproved quality of higher resolution pictures--that are not reallydiscernible at present distance between set and viewer, is desirable andpossibly only equally important to the viewer than the ability toreceive additional channels. Accordingly, it appears that a need existsto provide a higher quality picture as that now made available by HDTVwhile allowing the broadcaster to utilize to the greater extent, theremaining portion of the channel allocation for additional programmingor "data casting".

As to the first objective, which, if addressed properly, allows thesecond objective to be met, the history of TV evolution reveals theperennial problems needing solution to meet the resolution revolutionthat now requires brighter, smarter picture processing to live up toHDTV promises of 35 mm quality.

Naturally, the periphery of the picture is blurred by TV cameras thatpan, tilt and track to keep in the center of TV moving images that arecenter of attraction such as an ice skater or player with the football,basketball, puck etc. From empirical human factors data obtained from TVtrackers and "moving window" scan converters (Rand Corp., RCA Corp.,Fairchild Camera et al.) a target should move through the standard TVfield of view in 4 seconds or more but definitely no faster than onesecond. One second equals time for refreshing 60 fields and also 3.4fixations of 5×5 degrees at 0.3 second assuming 480 scan lines eachsubtend arc minute defined as 20/20 vision.

Thus those conclusions that an object ought not to move more than 1-to-4scan line in 1/60th second field is consistent with video compressionratios of 30-120 since 1.7%-0.42% or less is changed in either or bothaxes.

The prior art of monochrome video scan conversion was considered to be alinear system using analog and later digital means with predictableperformance. One of the earliest scan converters involved a TV camerascanning a radar PPI plan position indictor whereby the video values at"rho-theta" (range and angle) polar coordinates driving an oscilloscopedisplay with long persistence phosphor were transformed to the x,yCartesian coordinates of the horizontal and vertical deflection in TVcamera feeding a synchronized TV display. It and subsequent scanconverters also used to convert diagnostic ultrasound and sonar signalsto TV images mechanized the coordinate transformation matrix called theJacobian in Calculus texts.

Generally, when higher resolution ultrasound or radar indicators or TVrasters eg. 1225 lines EIA-RS-343 are scan converted to standard 525lines EIA-RS-170, 330 severe Moire patterns form by the beat-frequencybetween the horizontal line frequencies if the readout beam was notsufficiently defocussed to span several input scan lines. Preferably ananti-Moire digital filter or dither analogous to analog "wobble" signalcan modulate the scan converter input PPI deflection as a function ofrho, theta to widen the spokes as it were near the rim when nearlyparallel to horizontal scan lines readout.

A patent in the early 1970's granted to Don Dudley of Dataplex Inc.explained how 1225 line HDTV could be broadcast in place of redundantfields via standard 525 line TV. In principle, that is what HDTV doestoday in transmitting the differences between fields with more robustalgorithms for removal of redundancy subject to acceptable rates ofdistortion and the empirical 70% Kell factor or actually 2/pi=64%modulation transfer function (MTF) at Nyquist sample limit.

Another early scan converter involved the simplistic reduction of 625 TVlines 25 frames/second in Europe to 525 TV lines 30 frames/second in USAby cropping off 100 lines, including at times the forehead of thetelevised person. Subsequently, instead of cropping 50 lines from thetop and bottom, 4 of every 25 lines were dropped out or "decimated"since 625/525=25/21. Preferably, as DTV FSC demonstrated in 1985, foreach and every one of 21 readout lines falling between 2 of 25 linesweighting factors proportional to their separations enabled theinterpolation of video values between the 625 lines and smootherdecimation to 525 than simplistic drop outs. Likewise for example,720p*2/3=1080L*4/9=480.

In addition to spatial artifacts of aliasing from subsampling and ofcontouring from quantization in digital scan conversion, the differentframe rates created temporal problems with moving images analogous tothe stroboscopic effect of backward wagon wheels in sampling 25-30frames/second. The modern solution by Silicon Graphics et al. is knownas "morphing" is disadvantageous since its temporalinterpolation/decimation between fields would require brute-force RAM orexcessive software "bloatware" for MPEG-2 decoding.

Before digital random access memories (RAM) became inexpensive andcommonplace, instead of a TV camera tube scanning one CRT picture anddriving another CRT, Texas Instruments, Tektronix, Hughes Image Devicesand other companies produced scan converter tubes with electronic input(write) and output (read) video whose associated horizontal and verticaldeflection coils were independent in so called "double-ended" tubessimultaneous read and write. They were time-shared in cheaper"single-ended" tubes as in RAM. Most of these CRT scan converters wereeventually replaced by digital image processors or TV "frame grabbers"using RAM in a "brute-force" manner because of the Nyquist samplinglimit since RAM was so inexpensive.

Just as the Taylor Series representation in the time domain of the videosignal by summation of its time derivatives is the mathematical basisfor interpolation or reading video values between the scan lines so alsothe Fourier series representation in the frequency domain by summationof harmonics of complex sinusoids is the basis of video compression interms of DCT Discrete Cosine Transform that is the real part of theFourier Transform. Accordingly, the video transformation may bemechanized in either the time domain as in all known scan converters orin the frequency domain as will be discussed.

While in the past various approaches addressed changes in broadcastingformats and changes in the industry, presently what has not beenaddressed successfully is the ability to provide for enhanced quality TVresolution as provided for by HDTV and DTV in conventional TV whilereserving as much as possible or desired of the remainder of the channelallocation for other programming and/or ancillary data. This issomething that is desired by both the viewers and the broadcasters.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide the fullestfidelity, video data integrity of HDTV within the constraints ofstandard TV sets which is cost effective.

It is a further object to provide externally image quality enhancementseg. "sparkle" of broadcast formats for HDTV so as to optimize theflicker-free field of view or sparkling clarity of TV and provide forfull view of the uncropped HDTV picture or full use of present TVscreen.

It is also an object to provide buffer storage for refreshing ofcompressed images and for video transformation without having to decodeand again compress and re-encode the HDTV. In this regard, briefly, thepresent invention provides scan conversion in the frequency domain usingthe equivalent Fast Fourier Transforms (FFT) of time scaling, shiftingand of video derivatives for interpolation/decimation because N nodesare efficiently reduced from N×N operations to N×Ln(N) and because itexploits benefit of compression ratio which is proportional toresolution.

It is a further object to provide without objectionable aliasing orflicker remarkably higher resolution than is ordinarily expected byNyquist sampling theorem and averaging circuits. In this regard,briefly, the present invention effectively circumvents the luminanceblurring and chrominance cross talk at edges due to averaging or due tolow pass filters at half the sampling frequency and dithers spatialfilter sparkle to smooth flicker.

It is also an object to provide video transformation of DTVnoninterlaced, so called progressive scan to standard interlaced rasterformat and conversely transform interlaced fields to noninterlacedframes for PC monitors and printers.

It is a further object to provide enabling technology and local storagefor compact video router transformations to allow and to promoteindividually addressed video transmissions through quasi cross-pointswitches connecting any number of video sources to any number of videodestinations.

It is a yet further objective of versatility in expandability up to65K×65K pixels of MPEG so as to provide the sensation of realism.

Lastly, it is the ultimate object of the invention to provide forenhance quality TV resolution as that provided for by HDTV whilereserving the remainder of the channel allocation for additionalprograms and ancillary computer data.

The present invention provides for the foregoing objectives by providingfor a means to receive, decode and scan convert 1920×1080i or 1280×720pHDTV or other DTV format on standard TV sets while substantiallyenhancing display thereof. It provides for sampling and filtering of thehigher definition TV images for scan conversion without typicalcrosscoupling of chrominace and luminance so as to sparkle on lowerresolution TV sets while substantially maintaining the input detailsubject to viewers complex spatial/temporal flicker tolerance. It doesthis by a dithering of raster origin offsets in zigzag fashion within3×3 or 2×2 array of pixels for scan conversion into one pixel readout.This allows for an HDTV quality picture to be broadcasted on a formathaving a reduced bandwidth so as to allow the broadcaster to have agreater portion of the remaining channel for other programming whilstallowing a standard TV to utilize the HDTV or DTV format broadcasted.

The present invention also provides for renewable conditional access todemultiplex multicast programs and ancillary data compressed in packetswithin one DTV channel allocation. It enables standard TV sets to becomecompatible with DTV, HDTV and flexible formats of PC monitors and hardcopy printers. Moreover, its electronics may be incorporated withinnewer TV sets to minimize redundancy of hardware and its electronics maybe incorporated within a integrated circuit chip.

BRIEF DESCRIPTION OF THE DRAWINGS

Thus by the present invention its objects and advantages will berealized, the description of which should be taken in conjunction withthe drawings wherein:

FIG. 1 is a general block diagram of the present invention showingconventional "A/B" switch between Satellite or CABLE TV & FREE TVANTENNA feeding into VVTD STB (set-top-box) which feeds the TV viaoptional VCR, not yet digital or high definition. The finer HDTV"virtual screen" as it were behind the coarse TV screen illustrates howthe comparable 1920×1080i or 1280×720p actually "SPARKLE" using zig-zagordered raster dither of origin offset to exploit the space-timebandwidth of viewers eye-brain. Note the well established fixate rate of300×300 arc minutes/300 milliseconds integrates 18 progressive frames vs9 1080i frames. Thus the progressive HD's radical-2 improvement makesthe perceived resolution and signal/noise nearly identical for720p=1080i.

FIG. 2 illustrates a comparison of raster formats allowing interlace1920×1080i thru 3×3 filters or progressive 1280×720p thru 2×2 filtersplus Multicasting of 4 other ancillary TV data or scrolled datacastingwhich fit conveniently in the bottom (or top) 25% otherwise blankedbecause the HDTV 16/9 aspect is 25% shorter than standard TV height.

FIG. 3 is a detailed functional block diagram of the versatile videotransformation including an in situ CPU, renewable conditional accesskey, advertisers credit/coupons and by way of example the 2×2 and 3×3operations on 1280×720p and 1920×1080i HDTV which are two of the mostlikely of the 18 flexible formats of DTV processed by the VVTD.

FIG. 4 is a functional diagram of block 41 of FIG. 1 illustrating theorigin offsets of ordered raster dither providing the SPARKLE feature ofthe present invention for the two most likely HDTV rasters, 720p and1080i. It also presents a graphic representation of the 2 dimensionaldot-interlace (2Di) readout raster frame of 640×480 particularly for TVsand VCR's with about half the 6 MHz bandwidth for VGA. A radical-2improvement in 2Di effective resolution results on axis from the phasealternated 11.93 MHz offset sample clock.

FIG. 5 illustrates by simple example of interpolation and decimation ofthe present invention of HDTV 720p lines by 2/3 and also 1080i lines by4/9 to 480 lines when the viewer chooses to fill the full height of theTV with HDTV, knowing 25% of the HD picture width is necessarily croppedoff since the 16/9 HDTV aspect is 4/3rds wider than standard 4/3 aspect.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Turning now more particularly to the drawings, FIG. 1 shows the generalset-top-box of the present invention in this regard. The utility ofversatility is especially desireable because of the present lingeringdebate and uncertain future of 1920×1080 interlaced lines preferred bysome TV broadcasters over the comparable 1280×720 progressive scanfavored for action sports, multicasting, datacasting and by personalcomputer companies because of high speed low cost data networking.

The TV input signal may be received from either broadcast antenna 2,from cable TV 4 or satellite which is transformed and output viaoptional VCR 8 to the TV set 12 preferrably as video signal or onchannel 3 or 4 RF radio frequency signal (60-66 or 66-72 MHz). A cableTV or satellite box 6 may be provided and has its own RF tuner so it isalso an objective of this "set-top box" 10 to ultimately avoid theexpense of a third or fourth tuner in the system and thereby be evenlower cost.

The present invention is directed to converting digital and highdefinition TV to standard TV with fullest fidelity and image integritydesireable in accordance with ergonomic definition for 20/20 vision,i.e. a arc minute of resolution (291 microradian) and also in accordancewith economic balancing of broadcasters expensive new equipment byrenewable conditional access key and advertisers credit/coupons tooffset consumers fee expected to be substantially less than cable TVcharges.

HDTV<->Transform all 1280×720p into 640×360 of 525TV

DTV Multicasting and Datacasting of ancillary info

HDTV<->Transform all 1920×1080i into 640×360 of 525

ATSC other 16 formats, progressive & interlace

Full TV Channel Capacity 6 MHz (VGA) 640×480

Radical-2 Resolution 12 MHz 2Di Dot interlace

NTSC full color 4.2 MHz 448×336 (typical TV set)

NTSC full color 3.6 MHz 384×288 (typical VCR)

Monochrome (256 gray levels) also 320×240 dots

Simplest bilevel, black/white TV field 320×240 dots

The HDTV "virtual screen" lattice in FIG. 1 vibrates behind the coarseTV screen as it were so that comparable resolutions of 1920×1080i and1280×720p are squeezed through 3×3 and 2×2 filters as FIG. 4demonstrates. They "SPARKLE" via zig-zag ordered raster dither of originoffset by exploiting space-time bandwidth of viewers eye-brain. It isimportant to realize the well established fixate rate of 5×5 degrees per0.3 second permits 18 progressive frames to be integrated at 60 Hz vs 9interlace frames at 30 Hz and thus progressive HD's radical-2improvement makes the perceived resolution and signal/noise comparable,not to mention less motion artifacts with 720p and the ancillarydatacasting and multicasting of several programs.

Therefore the crisper, wider visual impact of FCC, ANSI/SMPTE, CCIR,ASTC, ISO, IEC specification of 1280×720p and 1920×1080i pictureelements when viewed where just resolveable suggests that HDTV screensmust be viewed from 1/2-1/3rd the present average distance or else be atleast 2-3 times as high as standard (NTSC) TV picture tubes. DTV'sreduced transmission impairments lets viewers come closer and itssubnanosecond sync jitter eliminates interline flicker as a preconditionfor SPARKLE function. The effective vibration or pseudo random ditheringof the high defintion "virtual screen" seen through the lower resolutionTV screen apertures is further discussed below.

A comparison of raster formats is set forth in FIG. 2. In this regardthe upper raster format 2A shows the proposed raster format for HDTV1920×1080i and 1280×720p illustrated for example on a 30 inch highscreen having a width which is 16/9 times the height and 33% wider thana standard TV set of equal height which is illustrated in FIG. 2B.

FIG. 2B shows two ways of fitting the HDTV signal into the standardsize, by either cropping the HDTV width or blanking 25% of standard TVpicture height. However cropping has the disadvantage of losing 25% ofthe picture and shrinking wastes 25% of the 480 standard raster lines orheight.

At the same viewing distance, the most noticeable difference between thenew TV & old is the wider HDTV screen, which is 16/9 times the height or4/3 times the old NTSC width which is 4/3 times the same height, whether1080 lines or 480 lines. As illustrated in FIG. 2B instead of droppingout 5 of every 9 HDTV lines per frame, the present inventioninterpolation and decimation into 4 NTSC lines interlaced between the 9HDTV of 2 fields is smoother and preferred if motion is not sensed andfull standard height is used.

In this regard and for purposes of comparison and by way of example, forH=100 units (cm), old width is 133.33 and new width overscans to 177.77.Hence 1/8th would be cropped off each side; only 1440 dots are displayedof the 1920 unblanked HDTV dots and 8/15 times the 1440 compresses to768 dots/width with dot clock 4 times the NTSC color subcarrier 3579545Hz. Conversely the full NTSC could be viewed within 3/4ths of the HDTVwidth provided the 480 NTSC lines are interpolated and expanded by 9/4to 1080 and the 768 NTSC dots are interpolated and expanded by 15/8.Simple doubling of NTSC lines and dots is cheaper but repeat line addsabsolutely no new information.

Shown separately in FIG. 2C, the full screen of HDTV is augmented by"Multicasting"/"Datacasting" which fits into the otherwise 120 blankedlines and offers distinct advantages over 1080i producing the sameperceived resolution. Almost 25% of the 720p transmission is specificiedto transmit ancilliary data which rate is comparable to 1.5 MHz T-1phone lines being about 25 times faster than 56 Kb/s modems. As shown inFIG. 2B to squeeze the entire HDTV 16/9 frame into the NTSC 4/3 aspectratio then 1/8th of the NTSC screen is typically blank on top andbottom. The 360 lines must somehow display 720p and 1080i. There aremany 2×2 and 3×3 spatial filters well known in the art to do this.

A 3×3 median filter compresses 1080i/3 to 360 and 1920/3 to 640. Themedian value of 3×3 arrays of dots typically replaces the center pixeland does not blur edges as a running average smoothing filter does. Inthis case however the median filter replaces all 9 pixels as seen in thereadout raster as one pixel for one field. In the next field the 3×3window center shifts as much as the origin offset so a new median valuereplaces the newer set of 9 pixels as one readout and so on. The ordereddither thereby lets the eye-brain do the "error diffusion" keeping thechrominance and luminance in tact, as opposed to averaging andinterpolation filters.

Alternatively for some images, to smooth contouring or perception ofquantizing noise the center of the 3×3 can be replaced advantageously bythe average of the 8 nearest neighbors and be represented as one readoutpixel. The subtraction of such a low pass averaging filter produces a socalled "Laplacian filter" that enhances edges upto a point ofobjectionable noise. Because that is in morphic resonance with a similarLaplacian filter deduced to be in the visual cortex, (which overshootsperfect steps of gray) the analogous effect of high pass edges and/orlow pass averaging can be obtained in the eye-brain itself byselectively representing one and only one of 9 pixels in each 3×3 arrayin each field. Considering all sequences of 9 pixels, FIG. 4 zig-zagpatterns show how jumping maximum distances accomplish error diffusion,whether a median, or mean or single pixel is representative of the 3×3depending on the picture content and subjective quality. Hence theversatile video is also made adaptive to the distribution of spatialfrequencies contained in the MPEG coded data by the Filter Select signalfrom the CPU 31.

With NTSC full standard 4.2 MHz video the resolution is only 448×336even though about 480 scan lines are unblanked. The present inventionsserves to overcome such limitations as will be discussed more fully. Tofurther enhance perceived resolution, a radical-2 or 41.4% boost by 2Ddot interlace is possible by offset sampling by 12 MHz clock phasealternate lines (PAL) as shown in the lower center of FIG. 4. As DeanWendland of University of Dortmund demonstrated, the square Fourier(frequency) spectra is effectively rotated to a diamond so resolution onaxis is improved by radical-2.

Notwithstanding the foregoing TV set limitations, the DTV transmissionand present invention scan converts similar 1280×720p and 1920×1080i byexploiting the space-time bandwidth in the visual perception of theviewer. Furthermore, because of the video compression of DTVtransmissions several programs can be compressed with the allocatedchannel in addition to or instead of the HDTV format. The advantageprovided by the present invention is the feature of allowing the HDTVsignal to be compressed while maintaining the free broadcast quality ofthe picture and also allowing renewable conditional access to themultiple broadcasts compressed into each channel. A nominal user feewould offset broadcast expenses and be compensated by TV advertiserscoupons or credit 29.

Turning now to the VVTD shown in FIG. 3, a functional detailed blockdiagram is shown for the set top box 10 of FIG. 1. In this regard the TVsignal may be from broadcast antenna 2, from cable TV 4 or satellitewhich usually output via VCR 8 to the TV set 12 a Channel 3 or 4 RFradio frequency signal (60-66 or 66-72 MHz) or video signal unlimited bythe tuner. However since DTV and HDTV VCR are not yet available the VVTDvideo or RF output feeds into the VCR to be taped or displayed live onthe TV screen. Note the satellite or cable TV box 6 also has its own RFtuner so it is an advantage of the "set-top-box" device 10 to avoid theexpense of a third or fourth tuner in the system and thereby be lowcost.

The Channel 3 or 4 RF receiver amplifier signal 20 is detected by 24 andtuned to a selected channel frequency 22 from which the video signal isdemodulated in 25 and 27 and sync is decoded in 26. The synchronizationdata signals for the GEN LOCK circuit 28 which is phase locked to thecrystal controlled CLOCK GENERATOR 30 for both writing and reading theRAM video content. This clocks the Write and Read H-Count 32 which arepreset by respective H-sync and likewise the V-Count 34 is preset by theV-sync as determined by the CPU/RAM Interface 33 and Versatile VideoReformatter 35. The Origin Offset 40 to dither the raster will becomeclear later on. Basically the RAM rows and columns are addressed bycounting down from preset values until zero reloads the appropriatepreset. To save RAM, the HDTV can be converted going into it, so only640×480 pixels need to be stored. However the full HDTV picture can beinexpensively captured in RAM so that it can be selectively readoutthrough the adaptive spatial filtering to preserve as much fidelty aspossible.

The digitized Luminance (Y) and color difference signals (Cb, Cr) arelatched in register 39. They are processed with the RAM Read-Mod-Writesignal thru the ALU 38 arithmetic logic unit as a function of modes fromCPU 31 and the prior (A) and present (B) values. The RAM 43 is sequencedthru the addresses of the pixel (H) 32 and line (V) counters 34 offseteach field +/-1 or 0 by 40. RAM output also feeds the Sparkle/DSPfilters 41 which consists of pixel and line delays to provide at onceall video values of each instant pixel and its neighbors. Lookup table(LUT) 50, preloaded through Multiplexor 51 from CPU 31, converts the newDTV luminance and chrominance to standard TV values to be generated forthe purpose of optimizing the adjustment of the TV brightness, contrast,color sharpness and tint. This ALU operation can be performed forexample by Texas Instruments 74181 or else the same LUT 50 could alsoincorporate all the ALU functions, including for example transformationto RGB, Y, I, Q, HSI Hue, Saturation, Intensity for displays other thanthe TV set. The RAM output is filtered by 41 whose functionalmechnanization is shown in FIG. 4.

The digital video output from the RAM 43 is refreshed at 60 Hz fieldrate and the digital to analog converter 44 feeds it to the RF modulator46 which mixes the chroma burst and sync signal before going out to theTV set thru the Input Output I/O RF switch 48. Note in the case of ananalog TV signal coming in, the I/O switch allows it to be fed directlyinto the TV set. This may be done automatically or by way of a manualswitching device if the user knows which signal is being transmitted.The switch may be an A/B switch in which the entire set-top-boxelectronics integrated into one Versatile Video Transformation Chip(VTC) might ultimately be housed.

In addition the DTV data/Not-video, from the Demod/Decoder 27, whichenables renewable conditional access key, is fed to the CPU 31. Thisancilliary data such as defined in ANSI/SMPTE 296M-1997 for 1280×720poffers almost the same channel capacity as dedicated 1.5 MHz phonelinesbecause 25% of the 6 MHz DTV spectrum is not picture content. Thedatacasting affords high speed, low cost downloading of information tothe CPU and displays or user output devices besides the printer andmodem shown connected to CPU 31. In addition, with the proper keydecoded, unscrambled multicasting of low cost TV programs offerscompetitive advantages between free TV and rising cable prices. It isestimated that economies of scale to convert DTV for a massive market of1.15 billion TV sets, even remaining about the homes with new HDTV sets,should enable the VVTD to be so affordable initially that it willencourage the buyers willingness to pay for the VVTD. In addition to thefree portion of the allocated channel, the remaining portion which willbe expanded due to the present invention allowing the broadcaster to usea format having a reduced bandwidth whilst maintaining an effect of HDTVquality, may be dedicated to subscription or pay programming. In such acase, the decoder can discriminate (see i.e. decision boxes 56) betweensuch programming and whether a user is a subscriber, much the same ascable or satellite user systems work.

In addition, production of standard sets will not stop immediately sothey could incorportate the VVTD inside instead of duplicating much ofthe set-top-box electronics.

Returning now to the methodology involved, conversion of 1920×1080i HDTVto fit properly within 360 lines of VGA 640×480 may be better understoodby considering the VVTD picture "fabric" and 3×3 median or mean (MOM)within Sparkle filter 41. Narenda in IEEE article had shown advantagesof speed and circuit simplicity in finding the median of the median andhe also pointed out how it differs from the true median and meanfilters. The video mean, median, mode and peak-to-peak average arespecifically different electronic video parameters but somewhat the samein unskewed statistics.

For example the mean video value replacing the center of a 3×3 # arrayis called the Laplacian operator that is employed in electronic imagingfor edge enhancement as reportedly in the visual cortex. The mean yieldsthe least "mean square error" that is generally preferred because it is"tractable" i.e. easy to manipulate but the human eye does not computemean square error as such. The 3×3 MOM exploits rationale for 1920×1080ibecause of simplicity of division by 3 and it keeps the integrity ofchromaticity with respective luminance.

Because the perceived brightness is a linear response to density(=-Log(Relative Brightness), the average of picture elements finer thanarc minute resolution defined as 20/20 vision involves simple, scalararitmetic. But the combination of HDTV picture elements chromaticity isa more complex, nonlinear vector process wherein mental attributes ofhue and saturation (purity) are represented by the vector angle andmagnitude respectively. Unlike averaging filters, the median retainsproper luminance and chrominance, retains edge sharpness and providesnoise smoothing without distortion and without streaking due to extremeexcursions of video amplitudes or peak-to-peak average.

Median filters replace the central pixel of a N×N array with the medianvalue of all values in the array. It can be shown that the median yieldsthe least expected absolute error. Its chrominance and luminance exactlyequals the middlemost value. More simply, the middle number of anordered set of samples is the median. N-random samples can be sorted orrecorded in hardware histogram as taught in U.S. Pat. No. 5,537,483. Theextreme values do not influence the median as they affect the mean valuebecause the median value of an image histogram is the x-abscissa graypoint at which the integral of probability density function of x, p(x)is half the total integral for all x. For each random x-value addressthe RAM increments by one its contents that is cleared after thecalculation. For each x value y is the number of times x occurred.Compute y/N=p(x). In 3×3 y>4 gives median x value else add up the CDFcumulative distribution and choose the mean or median to make thepicture smoother or sharper. Begin again at x=0 and find the closest xpoint where the integral of p(x) upto x is the CDF(x) and total CDF/2 xabcissa value is the median. This takes time and power.

Quicker median of median filter replaces the central pixel of a line ofpixels with the median value and then computes the median of all medianswhich replaces the center pixel. The MOM however maintains HD fullsplendor of 1920×1080 by replacing every 9 pixels in 3×3 array with themedian or mean.

The fabric of new TV frames created by the present invention must beadaptive to 18 flexible-format raster Standards. The picture fabric isoptimized by the VVTD so that the high definition pictures SPARKLE onthe old TV sets in the full splendor of all 1920×1080 HDTV pictureelements (pixels) woven together to unite the motionless interlaced (i)fields or to alternate the raster lines of the "progressive" (p)noninterlaced inputs.

The present invention enhances perceived resolution up to 4-9 times theVGA interlaced raster output that is necessarily shortened to 640×360because the new HDTV aspect ratio reduces the 480 line height 25% from3/4ths of the width to 9/16ths. In simplest terms, a unique 2×2 or 3×3array sequence of maximum displacement vectors therein dithers theraster origin offset in saccade-like jerks among the 4-9 pixelsluminance (Y) and chrominance (Cb,Cr) represented by one readout pixel.

After MPEG decompression, VVTD DSP filters perform unique 4:1 or 9:1video compression by scan converting down to 640×360. Another filterreplaces the center HDTV pixel of each 3×3 array with the median valueinstead of averaging or decimating by division by 3 in each axis. Inconjunction with the integration in space and time, the 3×3 median ofmedian image filter effectively smooths the higher definition TV withoutblurring the edges as nearest-neighbor averaging does and without jaggededges from decimated drop-outs and aliasing due to subsampling.Similarly the 2×2 filters convert the 1280×720i.

Since DTV jitter spec is less than a nanosecond, interline flicker fromimperfect interlace is imperceptible even upon close viewing such thatthe degree of SPARKLE selected depends on viewing distance and nonlineartolerance of complex flicker. Accordingly the CPU 31 sends Filter Selectto 41.

The saccade-like zigzag origin offset also provides supersampling offull resolution HDTV picture to be displayed in time and integrated inthe eye-brain. The pseudorandom dither sequence is ordered to minimizeartifacts i.e. error diffusion and maximize perceived resolution.Impairments due to cross coupling of chrominance and luminance byinterpolation and averaging are avoided by the VVTD maintaining theproper chrominance with the median's true luminance value or that of the4-9 supersamples.

Where TV or VCR does not have direct video input and the RF tunerbandwidth is only about half the 6 MHz full channel bandwidth requiredfor VGA, offset samples interlace fields of 320×180 in 2D into 640×360of 525 frame and provide a 41% enhancement by effectively rotating the2D Fourier frequency response square to diamond as shown by DeanWendland of University of Dortmund. Of course checkerboard pixel patternbecomes gray but that is very rare in real life. Minor flicker results.

The HDTV Raster 1920×1080i interlaced fields is illustrated separatelyand combined as the progressive 1280×720p frame in FIG. 4 wherein therow, column matrix address (r,c) in RAM corresponds to TV lines andpixel counters. To display HD 4-9 times more pixels on old sets yetminimize visible artifact, the ordered sequences of origin offsets areshown in FIG. 4. Each 2×2 or 3×3 block is thus vibrated or dithered andcompressed into one pixel that SPARKLES with 4-9 sequential values or4-9 medians for 4-9 offsets. Strong pixel to pixel correlation minimizesthe 4-9 median differences because mathematically it is the best valuethat minimizes expected absolute deviations seen by the eyes whereas theaverage value minimizes mean-square-error which eyes don't compute.Since the median of even number of values is defined as the average ofthe two middlemost elements, the chrominance distortion discussed belowcan be avoided in the 2×2 spatial filter for 1280×720p by selecting theexact chrominance and luminance pixel value closest to the median ormean value or the most probable (mode).

The means to accomplish DSP spatial filtering is well known in the artof image processing. All 9 pixels in the 3×3 array must be madeavailable simultaneously so a 2 Kilobyte SRAM is a convenient TV-linedelay when it is read out the next line before being rewritten. TwoD-latches clocked as a serial shift register of one byte into anotherforms the pixel delays as shown in FIG. 4. Obviously the 3×3 hardwarealso provides the subset 2×2 array of pixels simultaneously.

Of course design deficiencies and visual impairments within older TVsets will not be eliminated by the DTV or VVTD and in fact some maybecome more objectionable to some small fraction of viewers. To make theA/B comparison tests, the VVTD includes a switch 48 of FIG. 3 to selectthe old or new, i.e. old NTSC or new ATSC HDTV or DTV as previouslydiscussed.

Support for the interpolation/decimation used in 41 of FIG. 3 beyondNyquist sampling limits can be found in Fogel, L. J., "A Note of theSampling Theorem," IRE Trans. Info. Theory Mar. 1, 1955 p47-48) andvalid Taylor Series representation of any signal in terms of its timederivatives or Fourier Series in terms of sinewave frequency harmonics.

With regard to FIG. 5 which is merely two simple examples, the followinginterpolation/decimation is performed. Banding is likely if every otherNTSC line averages the 720p line above and below it. Curve fittingbeyond two lines using as much as 9th order Polynomials have beenreported. However, because of the disadvantages of interpolation anddecimation, the VVTD attempts to employ the simplest filtering thatrelies on and exploits tuned sensitivities in the visual perception.

Given f'(kT)=df(t)/dt at t=kT, (k=0,1,2, . . . ) the period of timebetween samples, Ts=1/Fs=<1/Fc. Consider 9→4 lines:

    ______________________________________                                        V0/9 V1/9   V2/9   V3/9 V4/9 V5/9 V6/9 V7/9 V8/9 V9/9                         *3/4  *1/4    *1/2  *1/2                                                                              *1/4  *3/4                                            +             +         +                                                     V0/4           V1/4        V2/4      V3/4      V4/4                           ______________________________________                                    

Note the V0/9 and V9/9 are the same as V0/4 and V4/4. Since V2/4 isequidistant between V4/9 and V5/9 it is set equal to their average,whereas the V1/4 and V3/4 are interpolated (GEOMTRICALLY weighted)values. Since 1/4=0.25 happens to be 1/4th up the gap of 1/9th=0.111 . .. between 0.222 . . . and 0.333 . . . it follows

    V(1/4)=3/4*V(2/9)+1/4*V(3/9).

Likewise 3/4=0.75 happens to be 3/4th up the gap between 0.666 . . . and0.777 . . . such that

    V(3/4)=3/4*V(7/9)+1/4*v(6/9).

Because V1/9 and V8/9 are totally ignored these two dropouts distort thedecimated picture elements. By using the differences between all the HDDTV samples and differences between those differences, the 2ndderivatives f"(kT) provide a much more accurate characterization of thesignals.

Given f"(kT)=d f(t)/dt at t=kT, Ts<(3/2) (1/Fc). And given m (many)derivatives with m-th derivative being the highest, then Ts=1/Fs=<(1/2)(m+1) (1/Fc).

The above analysis in the time domain is equivalent to Fourier analysisin the frequency domain and the real part of the so called Fast FourierTransform (FFT) provides the MPEG coefficients of the DCT DiscreteCosine Transform in 53 of FIG. 3 used in DTV compression. Theycharacterize and reconstruct video signals into 8×8 blocks, much ofwhich may be zeroed to minimize redundancy subject to acceptable levelof distortion. In general any series of video elements can berepresented mathematically by a polynomial curve that can fit the videooscilloscope waveform as precisely or practical as desired. For examplef'(kT) gives straight line approximation, f"(kT) fits a parabolic shape,f'"(kT) fits any cubic spline etc.

To the extent adjacent pixels are said to be correlated they can becompressed effectively (eg. MPEG-2, CCIT-Group 4) by removing redundancyusing adaptive "windowing" techniques. In the above example the sampleV2/4 could be comprised of variable fractions (a0 . . . a9) of all 9 HDDTV lines such as

    ______________________________________                                        V0/9 V1/9   V2/9   V3/9 V4/9 V5/9 V6/9 V7/9 V8/9 V9/9                         *a0  *a1    *a2    *a3  *a4  *a5  *a6  *a7  *a8  *a9                          =V2/4                                                                         ______________________________________                                    

but in practice 3×3, 5×5 or 7×7 windows aka spatial filters are goodenough and often are more easily implemented in real time as if spatialfrequencies of the video image are separable in the horizontal andvertical direction. Note 3×3 # nearest neighbor average is in morphicresonance with 2nd derivative Laplacian operator (del-squared ∇V) in thevisual cortex.

For NTSC only a common GEN LOCK SYNC CHIP runs at 4 times colorsubcarrier such that when divided down by 455 the 2Fh is produced whichproduces interlace by further division by 525. Although the pictureelements are not square the 768 dots per width correspond to a videobandwidth of 7.2 MHz or about twice VCR resolution.

It is often desireable to transform RGB, Y,I,Q or Y, Cb, Cr values thatare interdependent, to the subjective HSV Hue, Saturation, Value colorspace because the HSV attributes are supposedly independent. A red andgreen pixel should not produce a yellow as averaging would, except whenthe TV color triads are deliberately less than the arc minute angularsubtense defined as 20/20 vision.

The foregoing filtering effectiveness and operability is supported bythe following publications: Optimum Signal Processing by (Prof)Sophocles Orfanidis at Rutgers SUNJ, Discrete-Time Signal Processing byOppenheim & Schafer, IEEE Special Issue on Digital Image Processing July1972 and most recently Digital Television Fundamentals by Robin & Poulinand standards mentioned above.

Thus the present invention realizes the aforenoted objectives,advantages and features, and although the preferred embodiments havebeen disclosed and described in detail herein, its scope should not belimited thereby but rather its scope should be determined by theappended claims.

What is claimed is:
 1. A versatile video transformation device thatallows for conventional television sets to utilize input signals withHDTV, DTV or formats with enhanced data communications, compression, andimproved picture quality perception comprising:means for decoding saidsignal and scan converting them into an enhanced visual image fromstandard TV raster format and which comprises; means for spatial andtemporal filtering said input signals by adaptive sampling said inputsignals into selectable adjacent arrays, determining a selected valuefor each array, said selected value being selected from one of themedian, mean (average) of the input signals and a sample value of eacharray about a center pixel defined with respect to the raster originwhich is offset from a previous raster origin in the respective arrays,and displaying at least one single pixel in the array dependent on saidselected value; and wherein field sequential arrays are filtered andsaid selected value for each array that is used to readout a video valuein each array is determined based upon the decoded picture content,image motion, and respective origins that are offset with respect toeach other in an ordered dither fashion for error diffusion.
 2. Theinvention in accordance with claim 1 wherein the digital signal is inone of a 1920×1080i, 1280×720p HDTV format.
 3. The invention inaccordance with claim 1 wherein the scan conversion temporal and spatialfiltering is without color distorted crosscoupling of chrominance andluminance so as to sparkle on lower resolution TV sets whilesubstantially maintaining the input detail subject to viewers complexspatial/temporal flicker tolerance.
 4. The invention in accordance withclaim 1 wherein the ordered dithering of raster origin offsets isprovided in a zigzag fashion within a 2×2 array of filtered pixels forscan conversion into one pixel readout so as to sparkle on lowerresolution TV sets while substantially maintaining the input detailsubject to viewers complex spatial/temporal flicker tolerance.
 5. Theinvention in accordance with claim 1 wherein the ordered dithering ofraster origin offsets is provided in a zigzag fashion with 3×3 array of9 pixels displayed at one pixel so as to sparkle on lower resolution TVsets while substantially maintaining the input detail subject to viewerscomplex spatial/temporal flicker tolerance.
 6. The invention inaccordance with claim 1 which further comprises renewable conditionalaccess to demultiplex multicast programs and ancillary data compressedin packets within one DTV channel allocation.
 7. The invention inaccordance with claim 6 which includes means to indicate consumerreimbursement of the use fee of conditional access channels by one ofadvertisers' coupons and credit.
 8. A method of providing digital scanconversion for image signals having digital signal formats comprisingthe steps of:providing adaptive image acquisition means to minimizegrainy appearance of electronic noise frozen with the signal; providingadaptive thresholding of image subtraction means to detect image motionand thereby limit smearing; and providing digital versatile videospatial/temporal filtering means responsive to decoded picture parametersignals and automatically synchronized to formats of vertical andhorizontal sync signals associated with the input video and phase lockedto picture element clocks from a gen-lock circuit which also serve toclock counters addressing RAM random access memory input and read out.9. The method in accordance with claim 8 which includes providingPC-TV-DSP convergence means which enables standard TV sets to becomecompatible with DTV, HDTV multicasting, datacasting and formats of PCmonitors and hard copy printers.
 10. The method in accordance with claim8 which includes the step of incorporating electronics to perform themethod within TV sets to minimize redundancy of hardware and exploitframe memory required therein for image redundancy removal bycompression.
 11. The method in accordance with claim 8 which includesproviding an integrated circuit chip which performs the steps set forththerein.
 12. The invention in accordance with claim 1 wherein said videotransformation device is incorporated in an integrated circuit chip. 13.A method of transforming a digitally formatted television signal into aform utilizable by a conventional television comprising the stepsof:spatial and temporal filtering said digital TV signals by adaptivesampling said DTV into selectable adjacent arrays, determining aselected value for each array based on a sample value of the respectivearrays, the median, or mean (average) of the signals for each arrayabout a center pixel defined with respect to the raster origin which isoffset from a previous raster origin for displaying at least one singlepixel in the array based on said selected value; and providing fieldsequential arrays which are filtered and said selected value for eacharray that is used to provide a readout video value in each array whichis determined based upon the picture content, image motion, andrespective origins that are offset with respect to each other in anordered dither fashion for error diffusion.
 14. The method in accordancewith claim 13 wherein the digital signal is selected from signals havingone of a 1920×1080i and 1280×720p format.
 15. The method in accordancewith claim 13 wherein scan conversion temporal and spatial filtering isprovided without color distorted crosscoupling of chrominance andluminance so as to provide sparkle on lower resolution TV sets whilesubstantially maintaining the input detail subject to viewers complexspatial/temporal flicker tolerance.
 16. The method in accordance withclaim 13 wherein the ordered dithering of raster origin offsets isprovided in a zigzag fashion within a 2×2 array of filtered pixels forscan conversion into one pixel readout so as to provide sparkle on lowerresolution TV sets while substantially maintaining one of a 1280×720pand an input detail subject to viewers complex spatial/temporal flickertolerance.
 17. The method in accordance with claim 13 wherein theordered dithering of raster origin offsets is provided in a zigzagfashion with 3×3 array for median or mean filtering of 9 pixels into 9medians displayed at one pixel so as to provide sparkle on lowerresolution TV sets while substantially maintaining one of a 1920×1080iand an input detail subject to viewers complex spatia/temporal flickertolerance.
 18. The method in accordance with claim 13 which furthercomprises the step of providing renewable conditional access todemultiplex multicast programs and ancillary data compressed in packetswithin one DTV channel allocation.
 19. The method in accordance withclaim 13 which includes means to indicate consumer reimbursement of theuse fee of conditional access channels by one of advertisers' coupons orand credit.